[SPARK-61100][CONNECT][PYTHON] Preserve tuple semantics for Python Row - #383

What changes were proposed in this pull request?

Return one public tuple-based Row through the native extension and Python imports. Embed PySpark 4.3.0's Row implementation, preserve mutable __fields__, and materialize Rows for collection and iteration. Share field names per collected result, per top-level iterator Arrow batch, and per nested schema path.

Update local-input conversion, pickling and UDT conversion to use that identity. Add tuple, field-mutation, nested-input, batch and UDT/spatial regressions.

Why are the changes needed?

SPARK-61100. The native object wrapper lacks tuple inheritance and has different identity and field-mutation behavior, affecting tuple-aware serializers and pandas-on-Spark paths.

Includes the NumPy prerequisite (SPARK-61102) and binary configuration/iterator follow-up (SPARK-61099).

Does this PR introduce any user-facing change?

Yes. Rows support PySpark's tuple interface, public identity and field-name semantics. The inherited customized-pandas named-Row conversion behavior is retained; complete pandas parity is not established.

Tuple compatibility has a measured performance cost. For 100,000 collected rows followed by full recursive consumption, comparing the final binary parent with this change:

Paired CPU ratios were 1.31–1.86× and wall ratios 1.26–1.65×; retained RSS increased by 14–59 MiB. Collect-only CPU ratios were 6.18–16.29× because values now materialize before return. Wide iterator consumption improved; narrow and nested iteration remained slower. These costs are an explicit tradeoff for tuple compatibility.

How was this patch tested?

• Formatting, Rust core/API units, integration-test compilation with and without WASM, and native-extension Python suites: 2,291 passed, eight live-server skips.
• Twelve native live cases and all 36 setup/call/teardown reports passed with verified imports; 172 unrelated cases were deselected. Eight smoke actions covered nested Rows, JSON, pickle, shared names and local inputs. The NTZ test's nested subprocess had no separate import telemetry.
• Scoped official transport checks: 85 passed, 34 skipped, no retries; all 12 mixed Classic/Connect collection cases skipped. API/member checks confirmed tuple inheritance and inspected members, with annotation differences retained.
• The three offline files named by CI passed 2,363 cases with eight remote-only skips. Configured Python-shim coverage was 84.3% across 72 files; embedded Row and compiled Rust are excluded. A causal control reproduced NoSource without the synthetic Row omission. The workflow's 90/75 values select badge colors, not fail-under thresholds.
• Both paired benchmarks completed 54 fresh workers, 27 pairs and 162 validated observations each. Results above are medians of process medians; paired ratios use paired samples. This is one macOS/Python/dependency set with three pairs, no confidence intervals or server-CPU measurements. RSS is an endpoint observation, not allocation or leak evidence. Official Python/newest-master performance was not measured. Auxiliary profiling ended with a cleanup error and its timings are excluded.

The shared full and cluster runs executed the NumPy prerequisite binary. This branch has the same 43 selected raw-RPC source files and separate native-import/live validation. This is source-based reuse of transport evidence; those full and cluster runs did not execute this branch's binary. They use official Python plans and converters, with no per-child import attestation, and do not validate native high-level conversion, native ReleaseSession or native artifact upload.

The full run completed 87 normal files: 2,586 passed, 641 skipped, one deselected and no failures, errors or retries. Eleven files were entirely skipped. A separate advisory check of the existing metadata-column manifest entry still failed because the test catalog plugin was unavailable (one failure, 57 deselections). The cluster phase passed 15 invocations across four modules (ten substantive cases and five helpers), with 14 JVM-guarded skips and no failures or retries.

The original full run had a seeded-random failure on two threads. The unchanged test failed identically in official Python and both saved Rust transports on two threads, and passed in all three on its default four-thread fixture. The complete successful rerun used that fixture. Server logs retain deliberate negative-test diagnostics and shutdown races; successful direct shutdown was recorded, without final descendant-process telemetry.

Clippy remains advisory exit 101; its source diagnostics were assessed against the baseline. Linux, Windows and the newest-master combined tree were not runtime tested locally. These results establish neither full converter parity nor performance parity.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex

This pull request and its description were written by OpenAI Codex.